Skip to content

Conversation

Copy link
Contributor

Copilot AI commented May 26, 2025

This PR adds support for OpenAPI 3.x specifications as input files for automatic test generation. The implementation allows Doc Detective to directly read OpenAPI files (JSON/YAML) and transform operations into executable test specifications without manual configuration.

Features

OpenAPI Detection & Handling

  • Auto-detects OpenAPI 3.x files in .json, .yaml, or .yml formats
  • Automatically generates test specifications from OpenAPI operations
  • Maintains backward compatibility with existing test specifications

Safety Classification

  • GET and POST operations are considered safe by default
  • PUT, PATCH, DELETE, HEAD, OPTIONS operations are considered unsafe by default
  • Skips unsafe operations unless explicitly marked as safe

x-doc-detective Extension Support

  • Root-level extensions for default configuration across all operations
  • Operation-level extensions for per-operation customization
  • Supports all existing openApi schema properties
{
  "openapi": "3.0.0",
  "x-doc-detective": {
    "server": "https://testing.example.com",
    "validateSchema": true
  },
  "paths": {
    "/users/{id}": {
      "delete": {
        "x-doc-detective": {
          "safe": true,
          "before": ["getUser"],
          "after": ["getAllUsers"]
        }
      }
    }
  }
}

Dependency Management

  • before operations executed as prerequisite steps
  • after operations executed as cleanup steps
  • Dependencies referenced by operationId or path+method combination

Implementation

  • Added new utility functions in openapi.js for detection and transformation
  • Modified parseTests in utils.js to handle OpenAPI files
  • Added comprehensive test coverage with unit and integration tests

Example

Given an OpenAPI specification with a GET /users operation:

openapi: "3.0.0"
paths:
  /users:
    get:
      operationId: getUsers
      summary: Get all users
      responses:
        200:
          description: Success

Doc Detective will automatically generate a test:

{
  "id": "getUsers",
  "description": "Get all users",
  "steps": [
    {
      "action": "httpRequest",
      "openApi": {
        "operationId": "getUsers"
      }
    }
  ]
}

Fixes #3.

Warning

Firewall rules blocked me from connecting to one or more addresses

I tried to connect to the following addresses, but was blocked by firewall rules:

  • cdn.fwupd.org
    • Triggering command: /usr/bin/fwupdmgr refresh (dns block)

If you need me to access, download, or install something from one of these locations, you can either:


💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

Summary by CodeRabbit

  • New Features

    • Added support for importing and transforming OpenAPI 3.x specification files (JSON and YAML) into test specifications, including handling of custom extensions, configuration inheritance, and operation dependencies.
  • Tests

    • Introduced comprehensive test suites to verify OpenAPI integration, YAML parsing, and transformation logic for test generation.
  • Chores

    • Added sample OpenAPI 3.x specification files for testing various API structures and custom extension scenarios.

Copilot AI changed the title [WIP] OpenAPI input support Add OpenAPI 3.x Input Support May 26, 2025
Copilot AI requested a review from hawkeyexl May 26, 2025 17:42
Copilot finished work on behalf of hawkeyexl May 26, 2025 17:42
@hawkeyexl
Copy link
Contributor

Remove the safe property from the x-doc-detective object. An operation being safe should be inferred from the existence of the x-doc-detective object or based on our built-in safety assessment.

Copy link
Contributor Author

Copilot AI commented May 26, 2025

Remove the safe property from the x-doc-detective object. An operation being safe should be inferred from the existence of the x-doc-detective object or based on our built-in safety assessment.

I've removed the safe property from the x-doc-detective extension as requested. Now, operations are considered safe if they have an x-doc-detective object or if they use one of the methods considered safe by default (GET, HEAD, OPTIONS, POST). The changes have been implemented in commit e5f86a8.

Copilot finished work on behalf of hawkeyexl May 26, 2025 18:12
@hawkeyexl
Copy link
Contributor

Create an OpenAPI definition to test the implementation. If you already have, explain what you've done in detail.

Copy link
Contributor Author

Copilot AI commented May 26, 2025

Create an OpenAPI definition to test the implementation. If you already have, explain what you've done in detail.

I've added a comprehensive OpenAPI definition in YAML format to test the implementation in commit 1d6ec5b.

The implementation already had two JSON-based OpenAPI test files:

  1. test/openapi-test-example.json - A basic API with user operations
  2. test/openapi-test-with-extensions.json - An API showcasing x-doc-detective extensions

I've added a new YAML example (test/openapi-test-example.yaml) that tests:

  • YAML format parsing
  • Root-level x-doc-detective configuration
  • Operation-level extensions and overrides
  • Dependencies between operations (before/after)
  • Various HTTP methods (GET, POST, DELETE)
  • Path/query parameters, request and response bodies
  • Schema references and complex data structures

I've also created a dedicated integration test (src/openapi-yaml-integration.test.js) to verify all functionality works with the YAML format.

The tests pass successfully, demonstrating that the implementation properly supports both JSON and YAML OpenAPI specifications.

Copilot finished work on behalf of hawkeyexl May 26, 2025 19:13
@coderabbitai
Copy link
Contributor

coderabbitai bot commented Jun 11, 2025

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Walkthrough

OpenAPI 3.x integration has been added, enabling the system to detect, parse, and convert OpenAPI JSON/YAML files into Doc Detective test specifications. New utility functions and extensive test suites validate detection, transformation, configuration inheritance, extension handling, and dependency management for OpenAPI-based test generation.

Changes

File(s) Change Summary
src/openapi.js Added OpenAPI 3.x detection and transformation utilities, including helpers for config merging and dependency steps.
src/utils.js Integrated OpenAPI 3.x file detection and transformation into test parsing logic.
src/openapi-integration.test.js
src/openapi-yaml-integration.test.js
src/openapi.test.js
Added test suites for OpenAPI 3.x integration, YAML parsing, and utility function validation.
test/openapi-test-example.json
test/openapi-test-example.yaml
test/openapi-test-with-extensions.json
Added OpenAPI 3.0.0 specification files for users, pets, and products APIs with and without custom extensions.

Sequence Diagram(s)

sequenceDiagram
    participant User
    participant Utils
    participant OpenAPI
    participant TestSpecs

    User->>Utils: parseTests([filePaths])
    Utils->>Utils: Read file content
    alt isOpenApi3File(content, path)
        Utils->>OpenAPI: isOpenApi3File(content, path)
        OpenAPI-->>Utils: true
        Utils->>OpenAPI: transformOpenApiToSpec(doc, path, config)
        OpenAPI->>OpenAPI: extractOperations(doc)
        OpenAPI->>OpenAPI: transformOperationToTest(...)
        OpenAPI-->>Utils: Doc Detective spec object
        Utils->>TestSpecs: Add spec to array
    else not OpenAPI
        Utils->>TestSpecs: Standard test parsing
    end
    Utils-->>User: [specs]
Loading

Poem

In fields of YAML, JSON too,
The rabbits found a path brand new.
OpenAPI specs, now parsed with care,
Tests spring from endpoints everywhere!
Extensions guide, dependencies link,
Our burrow’s smarter than you think.
🐇✨


🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Explain this complex logic.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai explain this code block.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and explain its main purpose.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Support

Need help? Join our Discord community for assistance with any issues or questions.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

@hawkeyexl
Copy link
Contributor

@CodeRabbit review

@coderabbitai
Copy link
Contributor

coderabbitai bot commented Jun 11, 2025

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@hawkeyexl
Copy link
Contributor

@copilot Add detailed documentation about how OpenAPI documents are transformed into Doc Detective tests, including requirements for certain behaviors/methods, logic behind the workflows, and user expectations.

@hawkeyexl
Copy link
Contributor

@CodeRabbit full review

@coderabbitai
Copy link
Contributor

coderabbitai bot commented Jun 11, 2025

✅ Actions performed

Full review triggered.

Copy link
Contributor Author

Copilot AI commented Jun 11, 2025

@copilot Add detailed documentation about how OpenAPI documents are transformed into Doc Detective tests, including requirements for certain behaviors/methods, logic behind the workflows, and user exp...

I've added comprehensive documentation about OpenAPI transformation in commit a2512a8. The new documentation covers:

  • Detection and file support requirements
  • Complete transformation workflow and safety classification logic
  • x-doc-detective extension system with inheritance rules
  • Dependency management with before/after operations
  • Configuration examples and transformation workflows
  • Method-specific requirements and validation rules
  • User expectations, limitations, and best practices

The documentation explains how OpenAPI operations are transformed into Doc Detective tests, including the logic behind safety classification, extension processing, and dependency resolution.

Copilot finished work on behalf of hawkeyexl June 11, 2025 20:25
@hawkeyexl hawkeyexl marked this pull request as ready for review June 11, 2025 20:58
@hawkeyexl hawkeyexl requested a review from jstjoe June 11, 2025 20:58
Copy link

@jstjoe jstjoe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great. Test coverage and all. We should probably add to the test schema over time but this looks solid.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

OpenAPI input support

3 participants